193 research outputs found
Gaining insight from large data volumes with ease
Efficient handling of large data-volumes becomes a necessity in today's
world. It is driven by the desire to get more insight from the data and to gain
a better understanding of user trends which can be transformed into economic
incentives (profits, cost-reduction, various optimization of data workflows,
and pipelines). In this paper, we discuss how modern technologies are
transforming well established patterns in HEP communities. The new data insight
can be achieved by embracing Big Data tools for a variety of use-cases, from
analytics and monitoring to training Machine Learning models on a terabyte
scale. We provide concrete examples within context of the CMS experiment where
Big Data tools are already playing or would play a significant role in daily
operations
Higgs and B physics in Run II
In Run II at the Tevatron, the major goal of the upgraded CDF and \dzero
detectors is a Higgs search in the mass range of 110-200 GeV. They will also
contribute significantly to B physics. Among many possibilities they will be
able to measure rare decays of B mesons and improve our knowledge of CP
violation in B system through study of B mixing. Various aspects of Higgs and B
physics in Run II are discussed here.Comment: for the D0 Collaboratio
The archive solution for distributed workflow management agents of the CMS experiment at LHC
The CMS experiment at the CERN LHC developed the Workflow Management Archive
system to persistently store unstructured framework job report documents
produced by distributed workflow management agents. In this paper we present
its architecture, implementation, deployment, and integration with the CMS and
CERN computing infrastructures, such as central HDFS and Hadoop Spark cluster.
The system leverages modern technologies such as a document oriented database
and the Hadoop eco-system to provide the necessary flexibility to reliably
process, store, and aggregate (1M) documents on a daily basis. We
describe the data transformation, the short and long term storage layers, the
query language, along with the aggregation pipeline developed to visualize
various performance metrics to assist CMS data operators in assessing the
performance of the CMS computing system.Comment: This is a pre-print of an article published in Computing and Software
for Big Science. The final authenticated version is available online at:
https://doi.org/10.1007/s41781-018-0005-
Predicting dataset popularity for the CMS experiment
The CMS experiment at the LHC accelerator at CERN relies on its computing
infrastructure to stay at the frontier of High Energy Physics, searching for
new phenomena and making discoveries. Even though computing plays a significant
role in physics analysis we rarely use its data to predict the system behavior
itself. A basic information about computing resources, user activities and site
utilization can be really useful for improving the throughput of the system and
its management. In this paper, we discuss a first CMS analysis of dataset
popularity based on CMS meta-data which can be used as a model for dynamic data
placement and provide the foundation of data-driven approach for the CMS
computing infrastructure.Comment: Submitted to proceedings of 17th International workshop on Advanced
Computing and Analysis Techniques in physics research (ACAT
MLaaS4HEP: Machine Learning as a Service for HEP
Machine Learning (ML) will play a significant role in the success of the
upcoming High-Luminosity LHC (HL-LHC) program at CERN. An unprecedented amount
of data at the exascale will be collected by LHC experiments in the next
decade, and this effort will require novel approaches to train and use ML
models. In this paper, we discuss a Machine Learning as a Service pipeline for
HEP (MLaaS4HEP) which provides three independent layers: a data streaming layer
to read High-Energy Physics (HEP) data in their native ROOT data format; a data
training layer to train ML models using distributed ROOT files; a data
inference layer to serve predictions using pre-trained ML models via HTTP
protocol. Such modular design opens up the possibility to train data at large
scale by reading ROOT files from remote storage facilities, e.g. World-Wide LHC
Computing Grid (WLCG) infrastructure, and feed the data to the user's favorite
ML framework. The inference layer implemented as TensorFlow as a Service
(TFaaS) may provide an easy access to pre-trained ML models in existing
infrastructure and applications inside or outside of the HEP domain. In
particular, we demonstrate the usage of the MLaaS4HEP architecture for a
physics use-case, namely the Higgs analysis in CMS originally
performed using custom made Ntuples. We provide details on the training of the
ML model using distributed ROOT files, discuss the performance of the MLaaS and
TFaaS approaches for the selected physics analysis, and compare the results
with traditional methods.Comment: 16 pages, 10 figures, 2 tables, submitted to Computing and Software
for Big Science. arXiv admin note: text overlap with arXiv:1811.0449
The CMS monitoring infrastructure and applications
The globally distributed computing infrastructure required to cope with the
multi-petabytes datasets produced by the Compact Muon Solenoid (CMS) experiment
at the Large Hadron Collider (LHC) at CERN comprises several subsystems, such
as workload management, data management, data transfers, and submission of
users' and centrally managed production requests. The performance and status of
all subsystems must be constantly monitored to guarantee the efficient
operation of the whole infrastructure. Moreover, key metrics need to be tracked
to evaluate and study the system performance over time. The CMS monitoring
architecture allows both real-time and historical monitoring of a variety of
data sources and is based on scalable and open source solutions tailored to
satisfy the experiment's monitoring needs. We present the monitoring data flow
and software architecture for the CMS distributed computing applications. We
discuss the challenges, components, current achievements, and future
developments of the CMS monitoring infrastructure.Comment: 14 pages, 5 figures, submitted to Computing and Software for Big
Science, see https://www.springer.com/journal/4178
Prototype of a cloud native solution of Machine Learning as Service for HEP
To favor the usage of Machine Learning (ML) techniques in High-Energy Physics (HEP) analyses it would be useful to have a service allowing to perform the entire ML pipeline (in terms of reading the data, training a ML model, and serving predictions) directly using ROOT files of arbitrary size from local or remote distributed data sources. The MLaaS4HEP framework aims to provide such kind of solution. It was successfully validated with a CMS physics use case which gave important feedback about the needs of analysts. For instance, we introduced the possibility for the user to provide pre-processing operations, such as defining new branches and applying cuts. To provide a real service for the user and to integrate it into the INFN Cloud, we started working on MLaaS4HEP cloudification. This would allow to use cloud resources and to work in a distributed environment. In this work, we provide updates on this topic, and in particular, we discuss our first working prototype of the service. It includes an OAuth2 proxy server as authentication/authorization layer, a MLaaS4HEP server, an XRootD proxy server for enabling access to remote ROOT data, and the TensorFlow as a Service (TFaaS) service in charge of the inference phase. With this architecture the user is able to submit ML pipelines, after being authenticated and authorized, using local or remote ROOT files simply using HTTP call
Machine Learning as a Service for High Energy Physics on heterogeneous computing resources
Machine Learning (ML) techniques in the High-Energy Physics (HEP) domain are ubiquitous and will play a significant role also in the upcoming High-Luminosity LHC (HL-LHC) upgrade foreseen at CERN: a huge amount of data will be produced by LHC and collected by the ex- periments, facing challenges at the exascale. Despite ML models are successfully applied in many use-cases (online and offline reconstruction, particle identification, detector simulation, Monte Carlo generation, just to name a few) there is a constant seek for scalable, performant, and production-quality operations of ML-enabled workflows. In addition, the scenario is complicated by the gap among HEP physicists and ML experts, caused by the specificity of some parts of the HEP typical workflows and solutions, and by the difficulty to formulate HEP problems in a way that match the skills of the Computer Science (CS) and ML community and hence its potential ability to step in and help. Among other factors, one of the technical obstacles resides in the difference of data-formats used by ML-practitioners and physicists, where the former use mostly flat-format data representations while the latter use to store data in tree-based objects via the ROOT data format. Another obstacle to further development of ML techniques in HEP resides in the difficulty to secure the adequate computing resources for training and inference of ML models, in a scalable and transparent way in terms of CPU vs GPU vs TPU vs other resources, as well as local vs cloud resources. This yields a technical barrier that prevents a relatively large portion of HEP physicists from fully accessing the potential of ML-enabled systems for scientific research. In order to close this gap, a Machine Learning as a Service for HEP (MLaaS4HEP) solution is presented as a product of R&D activities within the CMS experiment. It offers a service that is capable to directly read ROOT-based data, use the ML solution provided by the user, and ultimately serve predictions by pre-trained ML models “as a service” accessible via HTTP protocol. This solution can be used by physicists or experts outside of HEP domain and it provides access to local or remote data storage without requiring any modification or integration with the experiment specific framework. Moreover, MLaaS4HEP is built with a modular design allowing independent resource allocation that opens up a possibility to train ML models on PB-size datasets remotely accessible from the WLCG sites without physically downloading data into local storage.
To prove the feasibility and utility of the MLaaS4HEP service with large datasets and thus be ready for the next future when an increase of data produced is expected, an exploration of different hardware resources is required. In particular, this work aims to provide the MLaaS4HEP service transparent access to heterogeneous resources, which opens up the usage of more powerful resources without requiring any effort from the user side during the access and use phase
Cloud native approach for Machine Learning as a Service for High Energy Physics
Nowadays Machine Learning (ML) techniques are widely adopted in many areas of High Energy Physics (HEP) and certainly will play a significant role also in the upcoming High-Luminosity LHC (HL-LHC) upgrade foreseen at CERN. A huge amount of data will be produced by LHC and collected by the experiments, facing challenges at the exascale.
Here, we present Machine Learning as a Service solution for HEP (MLaaS4HEP) to perform an entire ML pipeline (in terms of reading data, processing data, training ML models, serving predictions) in a completely model-agnostic fashion, directly using ROOT files of arbitrary size from local or distributed data sources.
With the new version of MLaaS4HEP code based on uproot4, we provide new features to improve users’ experience with the framework and their workflows, e.g. users can provide some preprocessing operations to be applied to ROOT data before starting the ML pipeline. Then our approach is extended to use local and cloud resources via HTTP proxy which allows physicists to submit their workflows using the HTTP protocol. We discuss how this pipeline could be enabled in the INFN Cloud Provider and what could be the final architecture
- …